Scaling collaboration with dbt Cloud from Coalesce 2023

"dbt cloud has a lot of new features that help us collaborate around data at scale. Since models are governed, we can trust them for analyses to inform our company."

- Cameron Afzal, dbt Labs

Product managers from dbt Labs, Jeremy Cohen and Cameron Afzal, discuss scaling collaboration in dbt Cloud. They share their experiences of navigating the complexities of scaling data infrastructure and introduced dbt Mesh, a new tool that enables domain-level ownership of data without compromising on governance or creating data silos.

Scaling collaboration in dbt Cloud is a key discussion in dbt's webinar

Jeremy and Cameron introduce the concept of scaling collaboration through new workflows in dbt Cloud. They used the analogy of a jaffle (or toasted sandwich) shop to explain the complexity of data management for a growing business, from managing suppliers, finances, and sales to dealing with an influx of data as the business scales.

Cameron shares, "As we expand to more data sources, contributors, and use cases, we're trying to serve thousands of employees who are hungry for insights. As we grow, our data complexity grows as well." Cameron expands on this, explaining the challenges of managing a large, centralized data project and the benefits of using smaller, separate projects. He points out that while individual developers might be happy with their own projects, the lack of a global, unified view of all data assets can be a problem.

Cameron and Jeremy then introduce dbt Mesh, a new approach they've found to be more effective for their data needs. This approach allows for domain-level ownership of data, from development to production, without compromising on governance or creating data silos. Jeremy explains, "dbt Mesh enables us to make data mesh a reality by offering a simple, cohesive way to integrate and manage data pipelines and products across the enterprise using a single data platform."

dbt Mesh offers a solution to balance governance and data silos

Jeremy and Cameron describe the innovative solution of dbt Mesh to address the challenges of data governance and silos. dbt Mesh, they explained, improves performance within individual projects and achieves the governance they were aiming for without sacrificing discoverability or creating data silos.

Jeremy points out, "dbt Mesh enables us to make data mesh a reality by offering a simple, cohesive way to integrate and manage data pipelines and products across the enterprise using a single data platform." He further explains that dbt Mesh allows for domain-level ownership of data.

"dbt Mesh enables domain-level ownership of data end-to-end, from development all the way through to production, without compromising on governance or creating data silos,” says Jeremy.

dbt Explore and Cloud CLI improve data visibility and flexibility

Cameron and Jeremy also introduce dbt Explorer and dbt Cloud CLI, new features in dbt Cloud that enhance data visibility and development flexibility. Cameron explains that dbt Explorer provides a shared canvas for collaboration between data developers and data analysts, showing the latest metadata after each dbt Cloud run.

"Explorer is our shared canvas with the data team. It's where we collaborate," Cameron states. He also emphasizes the importance of these features in providing access control and visibility for admins, ensuring secure and centrally governed data for large enterprises.

In terms of development flexibility, Cameron highlights the benefits of the dbt Cloud CLI: "The Cloud CLI enables our developers like Jeremy to build as they wish." Thus, these new features not only improve data visibility and understanding but also empower developers to work more efficiently and flexibly.

Cameron and Jeremy's key insights

• Scaling data infrastructure can be complex, particularly when transitioning from a small to a large organization
• One approach to managing data at scale is through dbt Mesh, which provides domain-level ownership of data end-to-end without compromising on governance or creating data silos
• dbt Mesh improves performance, discoverability, and governance, but requires a different way of working, which may not suit all organizations
• dbt Explorer is a tool that provides a map of your data pipeline, updating with the latest metadata after each dbt Cloud run
• Collaboration between data developers and data analysts is key to ensuring the right data sets are being developed and used